{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "04QgGZc9bF5D"
   },
   "source": [
    "# Classification using the Keras Sequential API\n",
    "\n",
    "**Learning Objectives**\n",
    "\n",
    "\n",
    "1. Build a neural network that classifies images.\n",
    "2. Train this neural network.\n",
    "3. Evaluate the accuracy of the model.\n",
    "\n",
    "\n",
    "## Introduction \n",
    "\n",
    "This short introduction uses [Keras](https://keras.io/), a high-level API to build and train models in TensoFlow.  In this lab, you Load and prepare the MNIST dataset, convert the samples from integers to floating-point numbers, build and train a neural network that classifies images and then evaluate the accuracy of the model.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/deepdive2/introduction_to_tensorflow/solutions/intro_logistic_regression_TF2.0.ipynb) -- try to complete that notebook first before reviewing this solution notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nnrWf3PCEzXL"
   },
   "source": [
    "## Load necessary libraries \n",
    "We will start by importing the necessary libraries for this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0trJmd6DjqBZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version:  2.6.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(\"TensorFlow version: \",tf.version.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7NAbSZiaoJ4z"
   },
   "source": [
    "Load and prepare the [MNIST dataset](http://yann.lecun.com/exdb/mnist/). Convert the samples from integers to floating-point numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7FP5258xjs-v"
   },
   "outputs": [],
   "source": [
    "mnist = tf.keras.datasets.mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BPZ68wASog_I"
   },
   "source": [
    "Build the `tf.keras.Sequential` model by stacking layers. Choose an optimizer and loss function for training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "h3IKyzTCDNGo"
   },
   "outputs": [],
   "source": [
    "# TODO 1\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dropout(0.2),\n",
    "  tf.keras.layers.Dense(10)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l2hiez2eIUz8"
   },
   "source": [
    "For each example the model returns a vector of \"[logits](https://developers.google.com/machine-learning/glossary#logits)\" or \"[log-odds](https://developers.google.com/machine-learning/glossary#log-odds)\" scores, one for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OeOrNdnkEEcR"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer flatten_2 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 0.06166657,  0.07144614, -0.07372011,  0.3451226 , -0.06205732,\n",
       "        -0.23894641, -0.00426888,  0.38629198,  0.11753443,  0.21888584]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = model(x_train[:1]).numpy()\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tgjhDQGcIniO"
   },
   "source": [
    "The `tf.nn.softmax` function converts these logits to \"probabilities\" for each class: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zWSRnQ0WI5eq"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.09631761, 0.09726418, 0.0841217 , 0.1278819 , 0.08510853,\n",
       "        0.07131012, 0.09017171, 0.1332566 , 0.10185182, 0.11271589]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.softmax(predictions).numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "he5u_okAYS4a"
   },
   "source": [
    "Note: It is possible to bake this `tf.nn.softmax` in as the activation function for the last layer of the network. While this can make the model output more directly interpretable, this approach is discouraged as it's impossible to\n",
    "provide an exact and numerically stable loss calculation for all models when using a softmax output. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hQyugpgRIyrA"
   },
   "source": [
    "The `losses.SparseCategoricalCrossentropy` loss takes a vector of logits and a `True` index and returns a scalar loss for each example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RSkzdv8MD0tT"
   },
   "outputs": [],
   "source": [
    "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) # TODO 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SfR4MsSDU880"
   },
   "source": [
    "This loss is equal to the negative log probability of the true class:\n",
    "It is zero if the model is sure of the correct class.\n",
    "\n",
    "This untrained model gives probabilities close to random (1/10 for each class), so the initial loss should be close to `-tf.log(1/10) ~= 2.3`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NJWqEVrrJ7ZB"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.6407173"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_fn(y_train[:1], predictions).numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9foNKHzTD2Vo"
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss=loss_fn,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ix4mEL65on-w"
   },
   "source": [
    "The `Model.fit` method adjusts the model parameters to minimize the loss: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "y7suUbJXVLqP"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 4s 74us/sample - loss: 0.2948 - accuracy: 0.9159\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 4s 68us/sample - loss: 0.1449 - accuracy: 0.9575\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 4s 67us/sample - loss: 0.1086 - accuracy: 0.9669\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 4s 67us/sample - loss: 0.0890 - accuracy: 0.9722\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 4s 67us/sample - loss: 0.0760 - accuracy: 0.9761\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb6b435e898>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4mDAAPFqVVgn"
   },
   "source": [
    "The `Model.evaluate` method checks the models performance, usually on a \"[Validation-set](https://developers.google.com/machine-learning/glossary#validation-set)\" or \"[Test-set](https://developers.google.com/machine-learning/glossary#test-set)\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "F7dTAzgHDUh7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 - 0s - loss: 0.0789 - accuracy: 0.9762\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.07894639570089057, 0.9762]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(x_test,  y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T4JfEh7kvx6m"
   },
   "source": [
    "The image classifier is now trained to ~98% accuracy on this dataset. To learn more, read the [TensorFlow tutorials](https://www.tensorflow.org/tutorials/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Aj8NrlzlJqDG"
   },
   "source": [
    "If you want your model to return a probability, you can wrap the trained model, and attach the softmax to it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rYb6DrEH0GMv"
   },
   "outputs": [],
   "source": [
    "probability_model = tf.keras.Sequential([\n",
    "  model,\n",
    "  tf.keras.layers.Softmax()\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cnqOZtUp1YR_"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(5, 10), dtype=float32, numpy=\n",
       "array([[4.16968859e-08, 6.23608756e-08, 5.28075470e-05, 2.82978057e-03,\n",
       "        1.16726082e-11, 1.71175685e-07, 3.04637446e-13, 9.97084558e-01,\n",
       "        4.06323215e-06, 2.85600290e-05],\n",
       "       [1.44917021e-08, 2.85007991e-05, 9.99968171e-01, 2.84016392e-06,\n",
       "        5.33308461e-15, 2.64569479e-07, 3.80620824e-09, 1.71865629e-11,\n",
       "        2.15737089e-07, 1.31360444e-14],\n",
       "       [4.27655920e-08, 9.99692202e-01, 3.80819870e-06, 4.24063728e-06,\n",
       "        1.78703067e-05, 1.56593114e-06, 4.08388269e-06, 2.54444923e-04,\n",
       "        2.10298167e-05, 6.26707333e-07],\n",
       "       [9.99903679e-01, 5.74413441e-08, 6.42219311e-05, 2.98295333e-07,\n",
       "        6.06606605e-08, 2.66647930e-05, 7.69448434e-07, 3.18793695e-06,\n",
       "        2.18482214e-07, 8.51092807e-07],\n",
       "       [5.69916847e-06, 1.11318251e-07, 1.25890219e-04, 7.61389970e-07,\n",
       "        9.85019803e-01, 1.97083864e-06, 1.77188417e-06, 1.21961719e-04,\n",
       "        3.18770253e-05, 1.46901319e-02]], dtype=float32)>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probability_model(x_test[:5])"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "rX8mhOLljYeM"
   ],
   "name": "beginner.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
